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Abstract 

We study the single production of fourth family quarks through the process pp Q'jX at the 
Large Hadron Collider (LHC). We have calculated the decay widths and branching ratios of the 
fourth family quarks {b' and t') in the mass range 300-800 GeV. The cross sections for the signal 
and background processes have been calculated in a Monte Carlo framework. It is shown that the 
LHC can discover single t' and b' quarks if the CKM matrix elements iVfql, |V^fe'| > 0.01. 
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I. INTRODUCTION 



The Standard Model (SM) of the electroweak and color interactions does not predict the 
number of the fermion families. However, due to the limits coming from strong coupling in 
QCD, the number of quark flavors should be less than eighteen. The electroweak precision 
measurements done by LEP experiments imply that the number of the light neutrinos is 
equal to three The replications of three quark and lepton families remain mystery as 
part of the flavor problem. A possible fourth family can play a crucial role to understand 
the flavor structure of the standard theory. The experiments at Tevatron have already 
constrained the masses of fourth family quarks. The CDF has the strongest bound on the 
mass oft' quark as mf > 258 GeV at 95% CL [2I. This analysis applies to t' pair production 
independent of the CKM matrix elements Vt'q. In particular, CDF obtains a bound on the 
mass of b' quark m^/ > 268 GeV |^ taking BR{b' bZ) = 1. Recent reexamination of 
the bounds on the masses of the fourth family quarks assumes mqi > 255 GeV [Jl- There 
exist some parameter space (mass-mixing angle) of the fourth family quarks which could 
be explored at future collider searches. The upcoming experiments at the Large Hadron 
Collider (LHC) are able to probe heavy quarks up to a heavier mass range accessible to 
future experiments. Recently, the pair production signals have been considered in Ref. ^ 
with a similar mass range what we consider here. Because of the uncertainties on the CKM 
matrix elements, there still remains an open door for the fourth family quarks and their 
mixings with the other three families. Recent electroweak precision S and T parameter 
analysis also supports to idea of the existence of the fourth family quarks [^. 

A prediction for the fourth family fermion masses as well as the CKM mixings has been 

un 

presented in [TL 18|. Pair production of the fourth SM family quarks has been studied at 
the LHC 0, Il0|. Recently, an update has been made on the analysis of the signal and 



background for the pair production of fourth family quarks within the ATLAS detector 

In this paper, we study the single production (less studied compared to pair production) 
of the fourth family quarks through the process pp b'{b')jX and pp t'(t')jX at the 
LHC. We have calculated the cross sections of signals and corresponding backgrounds in the 
Monte Carlo (MC) framework. The decay widths and branching ratios of the fourth family 
quarks (b' and f ) are calculated in the mass range 300-800 GeV. We analyze the potential 
of the LHC for probing the fourth family quarks providing three different parametrizations 
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of the CKM matrix elements \Vt'q\, \ Vqi,i 



II. INTERACTION LAGRANGIAN WITH FOURTH FAMILY QUARKS 

In the standard model of the electroweak interactions, quarks can couple to charged 
weak currents by the exchange of boson and neutral weak currents by Z° boson. The 
left-handed (right-handed) quark fields transform as doublets (singlets) under the group 
SU{2). We consider an enlarged framework of the SM to include fourth family quarks. The 
interaction of the fourth family quarks Q[{t', h') and the quarks Qi via the SM gauge bosons 
(7, 9, is given by 



Qi=b',t' 
Q'i=V,t' 

^ E Qa'{9'v-9\i')QK 



2cos9w 
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Q'i=b',t' 



(1) 



where Qe^g are electro-weak coupling constants, and gs is the QCD coupling constant. 
A^, G^, Z° and are the vector fields for photon, gluon, Z°-boson and W^^-boson, re- 
spectively. Qei is the electric charge of fourth family quarks; T"' are the Gellman matrices. 
gv and gA are the vector and axial- vector type couplings of the neutral weak current. Fi- 
nally, V — y^y' (where Y'^''^ being the Yukawa couplings) is the corresponding 4x4 CKM 
matrix: 



V 



(2) 



/ Vud Vus Vub Vub' \ 

Vcd Vcs Vcb Vcb' 

Vtd Vts Vt, Vty 

\ Vt'd Vt>b Vt'v ) 

The constraint on the mixing between the quarks comes from the unitarity of the CKM 
matrix in the SM. These constraints appear to be well satisfied experimentally for the three- 
family case and the elements of the 3x3 sub-matrix are well tested by various processes 



Jj. The elements in the fourth row and column are constrained by flavor physics, and the 
flavor changing neutral current (FCNC) efl'ects are suppressed by Glashow-Iliopoulos-Maiani 
(GIM) mechanism. A 4 x 4 quark mixing matrix V satisfy the unitarity conditions 

Ei^..p = Ei^./3r = i (3) 

a=l j=l 

for i,j = u,c,t,t' and = d,s,b,b'. In addition to these constraints, if the following 
constraints are also satisfied 

4 4 

a=l j=l 

for all indices (i 7^ j in the first sum and a 7^ /3 in the second sum) the triangles will 
have the same areas. First, second and third rows of the matrix V with the measurement 
from currently available experiments can be calculated as |\4b'P = 0.0008 ± 0.0011, \Vcb'\'^ = 
0.0295 ± 0.1780 and \Vtb'\'^ = 0.4054 ± 0.3696. For the first three columns we calculate 
|\4,rf|2 = -0.001 ±0.005, = 0.0315 ±0.1780 and = 0.4053 ± 0.3696. The mixing 

between the third and fourth family is less constrained. A lower limit can be obtained from 



the single production of top quarks, such as \Vtb\ > 0.68 at 95% C.L. [lj|. If there is a mass 



degeneracy between b' and t' quarks, the two body decay occurs mostly into third generation 
quarks via charged currents. 

III. PRODUCTION CROSS SECTION 

The fourth family quarks b' and f can be produced singly at the LHC. There are relevant 
processes qq Q'q', qq' Q'q" and qq' Q'q" for the single production as shown in Fig. 
[U As can be seen from Eq. (1) and Fig. [T] the single production cross section strongly 
depends on the CKM elements Vqb' or V^q. The cross sections for the subprocesses in Fig. 
[Dare summed to make "Q' + jet" analysis in the final state. The total cross section for the 
process pp Q'jX is expressed by 

dx 
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' r-»— Q' 1 ► i-^Q' 9^.^^ ^+ 

w+f_ w+f _ 7^ -* 

* '—^9' q' » '-^9" 9'^ ^9" 

(a) (b) (c) 



Figure 1: The diagrams for single production of fourth family quarks. 

where the lower limit of the first integral is chosen as ymin = ^q'/^- The parton distribution 
function (PDF) fg/p{x, Q^) is taken from the CTEQ6M with = m|, for the signal 
and Q'^ = s for the background. In order to compare the magnitude of single production 
cross section with the pair production we plot Fig. [2] depending on the mass of fourth 
family quarks. The pair production of fourth family quarks at the LHC is mainly due to 



gluon-gluon fusion and quark-antiquark annihilation processes [U]. In addition, there is 
also a contribution from the VT-boson exchange in the t-channel when the CKM matrix 
elements Vfg, Vqy are taken large, in which the relevant diagrams are shown in Fig. [3l We 
use three parametrizations called PI, PII and PHI for the CKM elements corresponding 
to Vq,, = V,Q, = 0.01, Vq,, = V,Q, = 0.05 and Kfe' = 0.044, V^' = 0.46, Vt^d = 0.063, 
Vfs = 0.46, Vfb = 0.47, respectively. The reason for choosing these options is that only the 
upper limits on the CKM elements could be well predicted from the current experimental 
data, and they can be relaxed as la over the average values. For a parametrization with 
equal strength ViQ' = Vqh, single production cross section becomes comparable to the pair 
production cross section for the value of CKM matrix elements Vj/g, Vq^ = 0.25 — 0.04 in the 
mass range mg/ = 300 — 800 GeV, as shown in Fig. [21 The cross section for single t' and 
b' production is almost the same for the parametrization PI and PII in the interested mass 
range. 

The decay channels of fourth family quarks are strongly related to the non-diagonal matrix 
elements Vfg, Vqy and the diagonal Vt'b'- The production cross section depending on the mass 
of fourth family quarks is given in Table [T] for the parametrizations PI-PIII. These values are 
obtained by using CalcHEP program In the first and second parametrizations the cross- 
sections are low as expected from the fact that single production cross section simply scales 
with |V^'gP and |V^6'p. All the related vertices are implemented into the program and the 
model files can be found at jisj]. In order to compare the range of the error in cross sections 
coming from the source of parton distributions CTEQ6L and CTEQ6M at = rriQ,^ we 
present the total cross section in Table [Til for single and pair production of fourth family 
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Figure 2: The cross sections for the single and pair production of b' and t' quarks at LHC for 
different parametrizations of the CKM matrix elements Vfg, Vqh> ■ 



Figure 3: The diagrams for the pair production of fourth family quarks through gluon-gluon fusion 
and quark anti-quark annihilation processes. 

quarks at the LHC. As can be seen from this table the relative difference in pair production is 
higher than in the single production mechanism. The single production cross sections differ 
by about 10% at uiqi = 300 GeV for two options of CTEQ distribution, while pair production 
differs at most ~ 18%. For higher mass values the difference between the two choices gets 
smaller. The ratios a{pp t' jX) / a{j)p ijX) and a{pp t'i' X) / a{j)p ttX) can also 
be used to reduce the uncertainties from the PDFs. The cross sections for the single and 
pair production of top quarks at the LHC are: 140.94 pb for single production and 492.36 
pb for pair production when the CTEQ6L is used at the scale = mf. 
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Table I: The cross section in pb for tlie single production of fourth family quarks b'{b') and t'{t') at 
the LHC. The parametrization PI-PIII are explained in the text. The numbers in the paranthesis 
denote the antiparticle case. Here, we have used CTEQ6M at the scale = itIq,. 





pp ^ b'{b')jX 


pp ^ t'{i')jX 


iVldoo 1 vjc V I 


PI 


PII 


PHI 


PI 


PII 


PHI 


300 


0.0617(0.0250) 


1.54(0.626) 


14.34(20.31) 


0.0631(0.0234) 


1.579(0.585) 


44.56(25.70) 


400 


0.0398(0.0149) 


0.995(0.372) 


7.95(11.77) 


0.0401(0.0134) 


1.003(0.335) 


26.06(14.44) 


500 


0.0273(0.0094) 


0.683(0.236) 


4.77(7.31) 


0.0272(0.0082) 


0.681(0.206) 


16.36(8.75) 


600 


0.0196(0.0063) 


0.489(0.157) 


3.03(4.78) 


0.0194(0.0053) 


0.484(0.133) 


10.82(5.59) 


700 


0.0145(0.0043) 


0.361(0.109) 


2.01(3.24) 


0.0142(0.0036) 


0.356(0.0897) 


7.42(3.71) 


800 


0.0109(0.0031) 


0.274(0.0772) 


1.38(2.26) 


0.0107(0.0025) 


0.268(0.0621) 


5.25(2.54) 



Table II: The total cross section in pb for the pair production of fourth family quarks Q' at the LHC. 
The values are given for two options of the parton distribution function CTEQ6 with = uiq,. 



g2 = m^, 


pp ^ Q'jX{Fl) 


pp ^ Q'Q'X{F1) 


Mass(GeV) 


CTEQ6L 


CTEQ6M 


CTEQ6L 


CTEQ6M 


300 


0.056 


0.062 


34.34 


40.36 


400 


0.036 


0.040 


7.63 


9.21 


500 


0.025 


0.027 


2.27 


2.77 


600 


0.018 


0.019 


0.82 


0.99 


700 


0.014 


0.014 


0.33 


0.40 


800 


0.011 


0.011 


0.15 


0.18 



IV. DECAY WIDTHS AND BRANCHINGS 

If rrit' > my , then t' and b' quarks can decay as 

t' W+b' 

W+b b' W-t (6) 

^ W+q{d,s) W-q{u,c) 
If my > mf , the b' and t' quarks can decay as 
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Table III: The branching ratio (%) for the h' decays according to the parametrization PHI. For the 
other parametrizations PI and PII the branchings change very small in the considered mass range. 



Mass(GeV) 


W-u 


W-c 


w-t 


300 


0.5 


56.3 


43.2 


400 


0.5 


50.2 


49.3 


500 


0.5 


50.3 


49.2 


600 


0.5 


48.9 


50.6 


700 


0.5 


48.8 


50.7 


800 


0.4 


48.8 


50.8 



y -> w-f 

W't t' W+b (7) 

^ W-q{u,c) W+q{d,s) 
In the first lines of the equations (l6|) and ((Tj), two body decay Q[ Q'j + takes place 
with a virtual (real) W boson depending on the mass difference \mt' — m^'l < 80 (> 80) GeV. 
In the analysis, both two body decays and three body decays (via virtual W^* boson) are 
considered. Fourth family quarks will decay into third family quarks associated with a 
boson as shown in the second line of the Eq. ^ and ((Tj). Last lines of the above equations 
denote the fourth family quark decays into light jets. 

The branching ratios for the b' decays according to the parametrization PHI are given 
in Table lllli As can be seen from this table, the fractions are dominant for b' — > W~c and 
b' W~t channels. For the other parametrization, the branchings change slightly in the 
interested mass range, for example: BR(6' W^ui^c)) = 36.6(33.4)% and BR(6' — > W^t) = 
26.9(33.3)% at m^/ = 300(800) GeV. Concerning the options PI and PII, the t' branchings 
BR(t' — > W~^q) = 33.3% remain the same for all channels in the considered mass range, 
while they are 51%(iy+6), A8%{W+s) and l%{W+d) for the option PHI. In Tablelmlthe b' 
decay modes and fractions are displayed. We have used these modes of fractions in order to 
calculate statistical significances (SS) as explained in the next section. 

In the case of mass degeneracy m^/ = mf,/ both t' and b' quarks decay into SM three family 
quarks with a charged current interaction. The total decay widths depending on the mass 
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Figure 4: Decay width of fourth family quarks for the parametrization PI-PIII. 

are given in Fig. [4] for three parametrization PI-PIII of the CKM matrix elements. The 
total widths are increasing smoothly with increasing masses. The option PHI has the highest 
total widths among the other options PI and PII. For example, at PHI with mqi = 700 GeV 
the decay width Fq/ = 1.5 GeV which is also comparable to that of the top quark. 

V. SIGNAL AND BACKGROUND 

The signal which we consider here for t' and b' single production include the decay chains 
t' W^b l^uib (or l^uij) and b' W^t l^uil^uib (or l^uil^uij), respectively. 
The anti-particle t' decays similarly but with an opposite charged lepton in the final state. 
However, b' or b' quarks decay eventually into two oppositely charged leptons, hence they 
should be taken into account together to enhance the statistics. Since a 6-jet can be identified 
with a good efficiency, we consider 6-jets in the final state. In our calculations we take into 
account leptonic decays of the W boson (W lui, where / = e,/i). The corresponding 
backgrounds contain a charged lepton, b-jet and/or other jets and missing energy for the 
single t' production signal; two oppositely charged leptons, b-jet and/or other jets and 
missing energy for the single b' production signal. We calculate the cross sections for the 
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W^Z, W+W-, ZZZ, ZZW^, W+W-Z, W+W-W^ and Whj, Wtj backgrounds. These 
backgrounds are calculated by using CalcHEP with the PDF CTEQ6M at = xhe 
results are given in Table HV] for the backgrounds including two or three weak bosons. These 
backgrounds give relatively small cross section when they are multiplied by the corresponding 
branching ratios for the interested channels. The backgrounds including Wbj and/or Wtj 
in the final state need to be studied in detail to see the signal over the background, while the 
others give relatively smaller cross sections. The background process pp WbjX includes 
the single production of top quarks as well. By applying an invariant mass cut on the 
mwb > 200 GeV this background can be reduced. The transverse momentum distribution 
of the final jet for the background processes pp — > WQjX is shown in Fig. El The pr 
distributions of the jet for W^hj and W~bj backgrounds have similar shape, while it differs 
from the backgrounds W^ij and W^tj. In order to reduce the relevant backgrounds and 
preserve the signal, we applied the acceptance cuts > 20 GeV for the final state jets. The 
invariant mass distribution mwQ is given in Fig. [H For the interested invariant mass range 
the backgrounds W^ij and W~tj contribute dominantly. We also apply an invariant mass 
cut as Irrit' — rn]y+i,\ < 10 — 20 GeV depending on the t' mass, and similarly, \mf,/ — m^y-^l < 
10 — 20 GeV for the b' signal. In this case, we obtain a significant reduction on the cross 
section of the background as shown in Table [Vl The reconstructed invariant mass spectrum, 
mwb, for the t' signal (where rrit' = 300 GeV and 600 GeV) and the corresponding background 
are given in Fig. [7l 

For single b' production, the main and most demanding background is W~tj process. 
We applied two different sets of criteria to reduce this background. In the first set, the 
transverse momentum cut > 20 GeV is applied. Applying the transverse momentum cut 
{p!p > 50 GeV) most of the soft jets can be rejected. The latter cut is more efficient than 
the first one, since with the former cut, some soft jets can still be found. Applying the strict 
cut (p^ > 50 GeV), reduction efficiency for the background W~tj is high enough for the 
interested mass range to observe the b' signal over the background. 

For the t' signal, we consider W'^Z, W~Z, W^W~ and W^bj reducible backgrounds. 
These are all reducible backgrounds respect to the applied cuts. The background cross 
sections with respect to the applied cuts are displayed in Table [Vl Here, the most obstinate 
background is W~^bj, which has the cross section of 240 pb before the cuts. It decreases to 
a reasonable value when we apply the invariant mass and transverse momentum cuts. The 
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Figure 5: The differential cross section depending on the transverse momentum of final state jet. 
The solid line (red) and dashed line (green) correspond to W'^bj and W~bj backgrounds, respec- 
tively; the dotted and dot-dashed lines refer to the backgrounds W~^tj and W~tj , respectively. 

Table IV: The cross-section in pb for the background relevant to the single b'{b') and t'{t') produc- 
tions as discussed in the text. The results are given for the backgrounds including two and three 
weak bosons in the final state. 



Background 


a{pb) 


ZZZ 


1.111 X 10^2 


ZZW+ 


2.050 X 10^2 


zzw- 


1.091 X 10^3 


W+W-W+ 


8.826 X 10^2 


w+w-w- 


4.463 X 10^2 


W+W'Z 


1.033 X 10-1 


w+z 


1.868 X 10^ 


w-z 


1.169 X 10^ 


w+w- 


8.367 X 10^ 
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Figure 6: Invariant mass distributions for W Q final state, the solid (red) and dashed (green) lines 
correspond to W~^bj and W~bj backgrounds, while dotted (blue) and dotted-dashed (purple) lines 
correspond to W~^tj and W~tj. 
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Figure 7: The invariant mass distribution for Wb signal from t' decay and the corresponding 
background. 
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Table V: The cross section in pb for pp W^tjX{W^tjX) and pp W^bjX{W^bjX) back- 
ground with (w) and without (o) cuts p?p > 20(50) GeV in the different mass range of Amwt or 
Am^Yf, (numbers shown in the first hne are in GeV) . 



Process 


no cut 


290 - 310 


390 - 410 


485 - 515 


585 - 615 


680 - 720 


780 - 820 


pp W+tjX 


9.771 


0.444(0.238) 


0.204(0.125) 


0.132(0.093) 


0.070(0.045) 


0.048(0.033) 


0.026(0.019) 


pp W~tjX 


11.385 


0.418(0.230) 


0.215(0.127) 


0.144(0.090) 


0.071(0.046) 


0.046(0.032) 


0.025(0.018) 


pp W^bjX 


240.342 


1.367(0.622) 


0.282(0.195) 


0.213(0.162) 


0.116(0.086) 


0.050(0.060) 


0.052(0.036) 


pp W^bjX 


180.294 


0.970(0.429) 


0.227(0.178) 


0.225(0.162) 


0.181(0.120) 


0.088(0.090) 


0.052(0.022) 



background W~bj is also important for t' signal. It should be considered together with the 
W^bj background, possible asymmetry for producing single t' and t' will give additional 
information. By applying the cuts > 50 GeV and choosing the appropriate invariant 
mass intervals Am = |mj/ — mwb\ the cross sections for the backgrounds can be reduced 
below the signal cross section level. 



VI. SIGNAL SIGNIFICANCE 



We define the t'{b') signal as the final state including a 6-jet or a light jet, a charged lepton 
(two leptons with opposite sign) and missing transverse momentum. We also require the 
reconstructed invariant mass distribution mwQ showing a peak around mg/ in the interval 
300 — 800 GeV. We can estimate statistical significance of the signal by using integrated 
luminosity of Li^t = lO^pb"^ that will be achieved in a year at the LHC run. By using 
integrated luminosity (Lint), signal (as) and background (a^) cross sections, the branchings 
BR{Q' WQ) followed by BR(14^ lu) to the chosen detectable channel, and the relevant 
efficiency e for the signal and background channel, we define statistical significance (SS) by 



SS = -^^/e-U^t (8) 



In our calculations, the 6— tagging efficiency (60%) and identification efficiency for each 
electron (90%) and muon (95%) are taken as reference values based on ATLAS studies [l^ 
for the relevant final states of signal and background. Significances are shown in Table [VT] 
with respect to the fourth family quark masses in the interval mg/ = 300 — 800 GeV for the 
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Table VI: Statistical significance (SS) for b'{b') and t'{t') at the LHC with the integrated luminosity 
of Lint = lO^pb-i. 



SS 


6'(&')-quark 


i'(t')-quark 


Mass(GeV) 


PI 


PII 


PHI 


PI 


PII 


PHI 


300 


1.73(0.69) 


43.22(17.27) 


640.91(892.34) 


2.94(1.31) 


73.67(32.86) 


3184.04(2211.22) 


400 


1.78(0.67) 


44.54(16.78) 


544.88(813.12) 


3.34(1.17) 


83.57(29.22) 


3325.72(1928.79) 


500 


1.49(0.51) 


37.45(12.73) 


396.28(597.43) 


2.49(0.75) 


62.26(18.83) 


2290.62(1225.12) 


600 


1.50(0.49) 


37.50(12.17) 


352.11(561.61) 


2.43(0.56) 


60.73(14.13) 


2079.25(909.39) 


700 


1.33(0.39) 


33.19(9.87) 


285.65(453.42) 


2.13(0.44) 


53.48(11.00) 


1707.09(696.92) 


800 


1.34(0.37) 


33.59(9.19) 


261.49(416.81) 


2.07(0.62) 


51.97(15.41) 


1559.32(965.05) 



three parametrization PI-PIII. In order to calculate signal significances for all mass ranges 
(300-800 GeV), one needs to calculate the branching modes and fractions, and associated 
total width for each mass values with proposed CKM options PI, PII and PHI. We use the 
branchings from Table [1111 for b' and almost constant values for t' together with the signal 
and corresponding background cross sections in the chosen invariant mass intervals. 

As can be seen from the Table IVll in the third option the statistical significances are very 
high while the other two options have fairly low values. The lowest SS values are obtained 
for the t' and b' productions in the PI option. Both 6'(6')and t'(t') signal can be probed at 
the LHC at the nominal luminosity with greater significances for the parametrization PII 
and PHI. 

As a result, this study shows that if the fourth family quarks are present, the LHC will 
discover them and measure their masses. The single production of fourth family quarks is 
very important to measure their mixings with the other families. For the PI-PIII options, t' 
signal will be observed with a significance greater than 2a if Vfq, Vqb' > 0.01 for interested 
mass range. As can be seen from Table IVTl the other options PII and PHI provide much 
better significances for the discovery of fourth family quarks in the mass interval 300 — 800 
GeV. 
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VII. CONCLUSIONS 



This study shows that in case of the single production of fourth family 6' and t' quarks, 
a mixing down to 1% can be measured at the LHC with nominal luminosity providing that 
the fourth family quarks have mass in the range 300 — 800 GeV. The measurements on the 
Vfq and Vqb' can also be improved if a possible luminosity upgrade is made at the LHC. In 
case the fourth family quarks are present, the LHC will discover them in pairs and measure 
their masses with a good accuracy. In addition, to obtain the mixing of the fourth family 
quarks with the other families, the single production of these new quarks remains to be 
measured. Once the LHC begins to run, the Higgs can be found using the golden mode 
[igI for the relevant mass range. Given the measurements of cross section for the Higgs 
production as well as the branching ratios of Higgs into possible decay modes, the LHC will 
be able to verify that the fourth family quarks do indeed exist. Combining the information 
from the pair production of fourth family quarks and their subsequent decays, the single 
production mechanism provide a unique measurement of the family mixing with the fourth 
family quarks. 
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